Automatic image enhancement

ABSTRACT

Approaches are described for managing the processing of image and/or video data captured by an electronic device. A user can capture an image using a camera of a computing device, where metadata obtained by sensor(s) of the device can be stored along with the image. The image can be transmitted to a network service, where the network service can divide the image into a plurality of image portions, and for each image portion, the network service can search a library of image patches in attempt to find at least one library patch that substantially matches a respective image portion. If one of the library image portions matches the image portion within an allowable threshold, the network service can modify the image portion such as by applying image modifications made to the library image patch to the image portion or merging the library patch image with the image portion.

BACKGROUND

As computing devices offer increasing processing capacity andfunctionality, users are able to operate these devices in an expandingvariety of ways. For example, mobile devices are increasingly offeringmultiple high quality cameras that are capable of capturing highresolution images and/or videos. However, in a number of situations,portable computing devices lack the computing power and/or storagecapacity to adequately process the images. Further, due to size,resources, and other limitations of conventional computing devices,although these devices can offer high quality cameras relative tocurrent mobile devices, cameras natively included on such devices aretypically lower in quality when compared to other designated cameras,such as digital cameras and single-lens reflex (SLR) cameras. The lowerquality of images produced by computing device cameras can reduce theoverall user experience associated with using computing devices.Accordingly, users are increasingly turning to network resources, suchas remote servers executing “in the cloud” to perform various tasks,such as to store and process data and process programs. As cloud-basedcomputing services continue to evolve and provide enhanced processingpower, greater storage, faster networks, and ubiquitous access to one'sdata, the utility to uses likewise increases.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments in accordance with the present disclosure will bedescribed with reference to the drawings, in which:

FIG. 1 illustrates an example environment in which aspects of thevarious embodiments can be utilized;

FIG. 2 illustrates an example system embodiment for enhancing an imagein accordance with an embodiment;

FIGS. 3( a), 3(b), 3(c), and 3(d) illustrate example image segmentationapproaches in accordance with various embodiments;

FIG. 4 illustrates an example group of patches useful for imageenhancement in accordance with an embodiment;

FIG. 5 illustrates an example process for enhancing an image inaccordance with an embodiment;

FIG. 6 illustrates an example device that can be used to implementaspects of the various embodiments;

FIG. 7 illustrates example components of a client device such as thatillustrated in FIG. 6; and

FIG. 8 illustrates an environment in which various embodiments can beimplemented.

DETAILED DESCRIPTION

Systems and methods in accordance with various embodiments of thepresent disclosure overcome one or more of the above-referenced andother deficiencies in conventional approaches to creating and/orprocessing image and/or video data. In particular, various embodimentsenable a network service (i.e., a network image service, image service,etc.) to enhance an image and/or video such as to improve a quality,resolution, sharpness, color depth, or other such aspect of the imageand/or video. For example, a user can capture an image using a camera ofa computing device (e.g., a mobile phone, tablet computer, etc.) orother electronic device, where data obtained by one or more sensor ofthe device can be stored as metadata along with the image. The metadatacan include positioning information, orientation information, and/ortemporal information, among other types of information. The image andmetadata can be uploaded to a network system or service, such as may beexecuting in a shared resource environment or “in the cloud,” forprocessing. In at least one embodiment, the network service can attemptto produce an enhanced image, such as to improve a quality, resolution,sharpness, color depth, and/or other such aspect of the image.

In at least one embodiment, a network service can perform aportion-by-portion analysis of the image (also referred to as the sourceimage), where the network service can divide or otherwise logicallysegregate the source image into a plurality of image portions. For eachimage portion, the network service can search a library of imageportions or “patches” in attempt to find at least one library imagepatch that substantially matches a respective image portion. As usedherein, an image patch refers to at least a portion of at least oneobject represented in an image having undergone at least one imagemodification. If one of the library image patches matches the respectiveimage portion within an allowable threshold, matching criterion, orother such parameter, the network service can modify the respectiveimage portion using the matching image patch. The modification caninclude, for example, determining image modifications that were made tothe library image patch and then applying one or more similarmodifications to the image portion. For example, if the sharpness,contrast, and hue were adjusted on the library image patch, the same orsimilar adjustment(s) can be made to the corresponding image portion.The network service (or another such system, component, or process) canrepeat this process of library image patch matching and image portionmodifying in order to generate an enhanced version of the source image.Thereafter, the enhanced and source image can be provided to the devicefrom the network service, where the user can be prompted to accept,save, and/or discard either image. Other approaches can be utilized aswell, such as to enable the subsequent viewing and utilizing of theenhanced image. The accepted (and in some instances the discarded)images can be stored on the computing device and/or on a database incommunication with the network service, among other such options.Various other functions and advantages are described and suggested belowas may be provided in accordance with the various embodiments.

As mentioned above, computing devices often include one or more camerasthat are capable of capturing images and/or videos. However, many ofthese devices lack the computing power and/or storage capacity toadequately process the images as compared to the processing capabilityof desktop or server-based systems. Further, due to size, resources, andother limitations of conventional computing devices, the camerasnatively included on such devices are typically lower in quality whencompared to other designated cameras. As such, it can be desirable toprovide access to more powerful resources that allow for computationallyintensive image processing applications directly from a user's computingdevice. Accordingly, since users are increasingly uploading image andvideo data from their computing device to remote servers, such ascloud-based photo albums having cloud-computing resources, theseresources can be used to perform image processing not supported by theuser's computing device. Accordingly, systems and methods in accordancewith various embodiments enable a network service (i.e., a network imageservice, image service, etc.) to enhance an image and/or video such asto improve a quality, resolution, sharpness, color depth, or other suchaspect of the image and/or video.

FIG. 1 illustrates an example environment in which aspects of thevarious embodiments can be utilized. As shown example environment 100 ofin FIG. 1, a portable computing device 102 is in communication with aremote server 120 executing across at least one network. The portablecomputing device includes, for example, a camera, an interface 104(e.g., a display element) that displays the field of view 106 of thecamera. The device can be aimed in different directions and theinterface can display an image or video of the current/active field ofview being captured by the camera. Although a portable computing device(e.g., a smart phone, an e-book reader, or tablet computer) is shown, itshould be understood that various other types of electronic devices thatare capable of determining and processing input can be used inaccordance with various embodiments discussed herein. These devices caninclude, for example, notebook computers; personal data assistants;cellular phones; video gaming consoles or controllers; portable mediaplayers; wearable computing devices such as smart watches, smartglasses; among others. The device can include other elements useful forimaging as well, such as a light sensor for determining an amount ofambient light and a white light LED, or other such illumination element,useful in illuminating objects within at least a portion of a field ofview of the camera. The remote server can be part of a shared resourceor multi-tenant environment, and can be any type of server such as anapplication server, a web server, etc.

A user of the computing device can use the camera to capture an image108 of a scene or subject matter(s) of interest 110. The image can betransmitted over a network 114 to a network service (e.g., networkedimage service) 116 operating on a remote server 120 for processing. Theimage can be transmitted along with metadata associated with the image(e.g., data tagged with the image, information related to the image,etc.). As described, the metadata can be obtained by sensor(s) of thedevice and can be stored along with the image.

For example, a global positioning sensor of the computing device candetermine information of where the image was captured and the locationinformation can be stored in metadata associated with the image. In thissituation, a global positioning system (GPS) chip, radio signalmultilateration system, cellular triangulation sensor, or other globalpositioning sensor of the device can be configured to determine ageolocation of the computing device at substantially the same time(e.g., within an allowable time period) as when the image was capturedby the camera of the computing device. In a further example, thecomputing device can present a user interface configured to receiveinput from the user of the device, such as to “tag” or otherwise specifya subject matter of at least a portion of the image by providing adescription of the portion of the image and the information can bestored in metadata associated with the image. In this situation, theuser can input data that indicates a subject matter included in theimage. For example, the user can specify a subject matter (e.g., nature,giant sequoia, etc.) for the entire image. In another example, the usercan specify a subject matter for a portion(s) of the image (e.g.,branch, leaves, etc.). In a further example, if a face of a person isincluded in the image, then an identifier or tag such as a name (e.g.,“Joe Smith”, “Jane Smith”, etc.) for the face or other aspect of theidentified person in the image can be included in the metadataassociated with the image.

It is further contemplated that a person of ordinary skill in art willrecognize various other information that can be included with and/orindicated by the metadata. For example, an orientation sensor(s) of thecomputing device can determine a direction at which a camera of thedevice is facing when an image is being captured by the camera of thedevice, and the orientation information can be stored in metadataassociated with the image. In this situation, the direction can bedetermined using at least one of a gyroscope, an accelerometer, acompass, among others of the device. In another example, the computingdevice can also utilize system settings (e.g., system clock/calendar) todetermine a temporal property (e.g., time of day, date, season, etc.) ofthe image when captured, and the temporal information can be stored inmetadata associated with the image. In this situation, the temporalaspect can include a time of day, a day of the week, a calendar date, aseason of the year, etc., determined using, for example, a systemclock/calendar of the device.

As described, the image and associated metadata can be transmitted fromthe computing device over the network (e.g., Internet, intranet, LAN,WLAN, etc.) to the network service. The network service can providenetwork infrastructure and resources for running various applicationsand/or other computing operations. The network service can communicatewith a database 130 that includes one or more image and/or patches ofimages. The images can include entire scenes and/or objects, or portionsof those scenes or objects. In accordance with various embodiments, theimages can be user provided images, images obtained from one or moresocial networks, images obtained from an online service, images obtainedfrom a digital image aggregator service and/or application, etc. Theimages can include images of objects, scenes, people, etc. In someinstances, the images may be high-quality images. Factors that cancontribute to the quality of the image can include lens type, camerasensor type (e.g., camera resolution), image processor component, andphotograph skill level of the photographer. Additionally oralternatively, the images can include images that have been enhancedthrough an image enhancement processes. This can include, for example,adjusting each of the images (or portions of the images) by one or morehuman image enhancement experts, or other image editing processes. Invarious embodiments, this can include using a panel of human experts tomodify one or more image characteristics of the images, whereinadjusting the image can include adjusting the entire image or portionsthereof. This can also include non-experts such as any person (orprocess) capable of editing an image. The images can be segmented into aplurality of patches of a same or differing size and these patches canbe improved by human experts (or non-experts). The adjusted patches canbe linked to the original unimproved patches of the same image.Additionally, in many embodiments, the procedure by which the patcheswere modified can be stored in the database.

In accordance with various embodiments, the image captured by the devicecan be provided to the network service in a number of ways. For example,users of the service can voluntarily provide images such as byindicating through a service preference or other preference that imagesuploaded to the service (or a subset of the images uploaded to theservice) can be used for such purposes. Additionally or alternatively,the service can use images obtained in-house, publically availableimages, images obtained for social networks, among others.

As describe, the network service can be configured to process the imageto generate an enhanced version of the image. In some embodiments, thenetwork service can provide the user (e.g., via the user's computingdevice) with access to the enhanced image version. For example, thenetwork service can transmit the enhanced image back to the user and/orthe user's device. In another example, the network service can provide alink to the user or user's device for accessing the enhanced image.Similarly, the user can access an account or network service, such as anetwork web photo album to access images enhanced by the service.

FIG. 2 illustrates an example implementation for creating and/orprocessing an image in accordance with an embodiment. As shown in FIG.2, a remote server 220 includes an image library 230 and at least oneimage 208 captured by a computing device that has been transmitted tothe remote server. Also transmitted with the image can be metadata(e.g., data tagged with the image, an image property, etc.). Themetadata related to the image can be obtained and/or generated via thecomputing device. As mentioned above, the metadata can include (but isnot limited to) geolocation information at substantially where the imagewas captured, a direction at which the camera of the device was pointingwhile capturing the image, a time of day/year when the image wascaptured, data inputted by the user to provide one or more details aboutthe image, and/or other information related to the image.

Operating on the remote server is a network service (not shown)configured to enhance the image. The network service can receive theimage and the metadata and can process the image in attempt to generatean enhanced version of the image. For example, the network service canvirtually divide the received image into one or more image portions 218.For example, the network service can select an image portion size (e.g.,5×5 pixels, 10×10 pixels, etc.) and can analyze the image with respectto a plurality of image portions at the selected size. In someembodiments, the network service can then create a copy of each imageportion and process each copy of the image portion such that theoriginal image does not get unintentionally modified.

In accordance with various embodiments, the image patch library can bean electronic storage unit configured to store a plurality of imagepatches. In some embodiments, the image patch library can reside withthe network service, while in other embodiments the image patch librarycan reside at one or more servers external to the network service. Asdescribed, the images can include entire scenes and/or objects, orportions of those scenes or objects. The images can be images that havebeen enhanced through an image enhancement processes. This can include,for example, adjusting each of the images (or portions of the images) byone or more human experts or other automated image editing processes. Invarious embodiments, this can include using a panel of human experts tomodify one or more image characteristics of the images. For example, theimages can be segmented into a plurality of patches of a same ordiffering size and these patches can be improved by human experts. Theadjusted patches can be linked to the original unimproved sections ofthe same image. Additionally, in many embodiments, the procedure bywhich the sections were modified can be stored in the database.

In some embodiments, each image patch can be part of a respective imagepatch group. For example, each image patch group can include multipleversions of the same image, where each version has undergone a differentenhancement. Additionally or alternatively, the image patch group caninclude related or similar images, where the image can be of the sameobject captured from different viewpoints. Additionally, the image patchgroup can include similar images or images that share at least one ormore common features. Further still, the image patch group can include alower resolution image patch and a higher resolution image patch. Assuch, the image patch group in this example can be an image patch paircomprising a “lo-res” patch and a “hi-res” patch (also referred to as a“lo-res-hi-res pair”).

The network service can communicate with the image patch library toperform image patch matching. In some embodiments, image patch/portionmatching can include comparing one or more pixels of a first imageportion (e.g., source image portion) to one or more pixels of a secondimage patch (e.g., library image patch) to, for example, determine asimilarity in at least one image aspect (e.g., pixel color, pixel hue,pixel brightness, etc.). For example, a set of pixels of the first imageportion can substantially match or correspond to a set of pixels of thesecond image patch if the two sets are sufficiently similar (e.g.,within an allowable deviation) with respect to one or more pixel aspects(e.g., pixel color, pixel hue, pixel brightness, etc.). In someembodiments, image patch/portion matching can be performed utilizing, atleast in part, line detection, edge detection, feature detection, imagecolor comparison, image texture comparison, image shape comparison,image distance measurement, and/or other image processing techniques.

Further, a number of other approaches can be used to determine howsimilar two images or image patches/portions are. For example, the levelof similarity can be determined using a correlation algorithm (such as across-correlation algorithm). For example, in one instance, the integercomponents of an image portion's brightness at each pixel can be used ina cross-correlation algorithm to determine the level of similaritybetween the image portion and the image patch. In another instance, eachpixel can be treated as a vector (e.g., in the case of color images),the vectors can be normalized, for example, to account for at least somevariation in brightness due to light and exposure conditions, and thecross-correlation between the two images can be computed. The result canbe a value between 0 and 1, and the result can be used to indicate apercent similarity of the two images. The result can be compared to apredetermined threshold, where in some embodiments, the thresholdindicates a level of acceptable similarity. For example, a result above85 percent can be used to indicate that the images are similar.

Further still, the level of similarity can be determined usingperceptual hash algorithms. Perceptual hash algorithms describe a classof comparable hash functions, and one or more features in an image canbe used to generate a distinct (but not unique) fingerprint, and thesefingerprints can be compared. For example, the fingerprints determinedin the image portion can be compared to the fingerprints of the imagepatch to determine the level of similarity between the images. It shouldbe noted that it is well understood how to determine a level similaritybetween images and/or image patch/portions, and that in accordance withan embodiment, various algorithms can be used to determine how similarone image is to another image, such as perceptual hash algorithms orother comparable hash functions, key point matching algorithms,histogram matching algorithms, among others.

For one or more of the image portions for the image, the network servicecan search for one or more unimproved patches that substantially matches(i.e., matches within an allowable threshold) a respective imageportion. If a matching unimproved patch is found for a respective imageportion, the respective image portion can be modified using an enhancedpatch that corresponds to the matching unimproved patch (i.e., theenhanced patch is in the same patch group/pair as the unimproved patch).In some embodiments, the respective image portion can be modified using(e.g., replaced by, merged with, etc.) the enhanced patch correspondingto the unimproved patch that matches the respective image portion. Inone example, merging can include comparing a set of pixels in theenhanced patch with a corresponding set of pixels in respective imageportion and determining middle (e.g., median, mean, somewhere inbetween, etc.) values (e.g., pixel color values) between the pixels inthe enhanced patch and the corresponding pixels in the respective imageportion. In another example, merging can include combining an area ofthe enhanced patch with a related (e.g., adjacent, nearby, etc.) area ofthe respective image portion which has no corresponding enhanced patch.

In other embodiments, rather than use the enhanced patch from thedatabase, the network service can use stored procedure information fromthe database for how the enhanced patch was modified and apply thatprocedure to the image portion. The procedure can be applied to theimage portion based at least in part on how well the image portionmatched the unimproved patch. For example, if the patches have asimilarity above a determined threshold, substantially the sameprocedure can be applied. In this way, the procedure can be appliedbased on the relative match between the image portion and the databasepatch. In other instances, the procedure method can be performedstochastically where the procedure can be modulated by image propertiesof neighboring image portions such that similar procedures are appliedto neighboring image portions or the procedure applied to the targetportion is modified to blend with the neighboring portions. For example,the modifications applied to a portion can be blended across one or moreneighboring portions such that each region of portions includes some setof filters and color correction. In this instance, the amount ofmodifications performed on neighboring portions can be based at least inpart on how similar the neighboring portions are to the database patch.In this situation, if two neighboring portions are very different, thetype and intensity of modification might diverge from what is applied tothe target portion.

Further, each filter can include one or more rules indicating how therule is applied based on the portion to which the filter will beapplied. For example, a filter can include a rule to apply a smoothingalgorithm to the image portion and neighboring image portions if thelevel of brightness of the target portion is greater than a thresholdlevel of brightness to its neighboring image portions. Additionally, arule can limit the amount of brightness adjusted for a particularportion if, for example, the level of brightness of the target portionis greater than a threshold level of brightness to its neighboring imageportions.

Various embodiments of the present disclosure can enable the networkservice to take into consideration the metadata related to the image andselect a subset of image patches in the library such that the subsetcomprises image patches that are associated with the metadata. In oneexample, if the metadata indicates that the image was captured at theGrand Canyon, then the subset can be selected to include only thoseimage patches that are associated with the Grand Canyon (e.g., onlyimage patches that are part of images that were captured at the GrandCanyon). As such, the utilization of the metadata related to the imageto select the subset can significantly improve the quality, accuracy, aswell as speed of image patch matching because the subset can be chosento be more narrowed/limited, but also more relevant.

In one example, the network service can perform patch/portion matchingfor a particular image portion of the image. The network service cancommunicate with the image patch library to select the relevant subset222 of image patches based on the metadata related to the image. In thisexample, the metadata can indicate that the image (and/or image portion)is of one or more trees. Accordingly, the subset can be selected toinclude image patches that are associated with trees or plant life. Thesubset can be searched in attempt to identify one or more image patchesthat substantially match the particular image portion.

The service can determine an image patch, from the subset, thatsubstantially matches the image portion of the image. If image patch isof a better image quality than image portion, then the patch can be usedto modify (e.g., replace, merge with, etc.) the image portion. In someembodiments, if image patch is associated with an enhanced version(e.g., if image patch is in an image patch group/pair that contains anenhanced patch that corresponds to image patch), then the enhancedversion patch can be used to modify (e.g., replace, merge with, etc.)the image portion. This process can be repeated one or more of the otherimage portions of the image.

In some embodiments, the network service can attempt to determinematching patches for smaller image portions and then progress to largerimage portions. For example, the service can first select the size ofimage portions for the image to be smaller (e.g., 5×5 pixels), performpatch matching for the smaller image portions, select a larger imageportion size (e.g., 10×10 pixels), then perform another patch matchingfor the larger image portions, and so forth.

Further, in some embodiments, the metadata can be used to apply one ormore automatic corrections to the image. For example, some cameras leaveartifacts 250 or other errors in the image due to how the lens andpixels interact. This can include, for example, color tint and sharpnessissues, among others. The metadata can include information such ascamera model and lens information and this information can be used bythe network service to apply one or more image processes to correct forsuch errors specific to the camera and lens information.

The enhanced image 260 can be provided to the computing device from theremote server, and the enhanced and/or the original image can bepresented to the user. The user can be presented a prompt on whether toaccept, save, or discard either image. The accepted (and in someinstances the discarded) images can be stored on the computing deviceand/or the remote server, such as in a database located on or separatefrom the remote server. In accordance with an embodiment, the imagesstored on the remote server can be made accessible to the computingdevice. For example, the user may have created an account with a servicethat provides a cloud-based photo album, and the user can decide tostore certain images in the cloud-based photo album.

As described, in various embodiments, the network service can beconfigured to divide the received image into one or more image portionsof any shape, size, and/or orientation. For example, FIGS. 3( a), 3(b),3(c), and 3(d) illustrate different approaches that can be used todivide the image into one or more image portions. As illustrated inexample 300 of FIG. 3( a), the network service can segment the imageinto square portions. The square portions can be any size, such as a 5×5pixel square, a 10×10 pixel square, among others. The database caninclude similar sized patches, both unimproved and human improvedpatches, and the image portions can be matched to the unimproved patchesin the database. Further, other shapes and sizes can be used inaccordance with various embodiments, such as rectangles, circles, etc.For example, example 300 of FIG. 3( b) illustrates the situation wherethe network service segmented the image into triangle portions. Asdescribed in regard to the square portions, the triangle portions canvary in size. Further, the database can include similar sized patches(e.g., both unimproved and human improved patches), and the networkservice can match the image portions to the unimproved patches in thedatabase. In accordance with other embodiments, the image can besegmented into different regions or shapes, such as one of the periodicshapes or patterns described in FIG. 3( a) or 3(b), or an irregularshape or pattern, such as that illustrated in example 340 of FIG. 3( c).In this example, the image can be segmented into one or more shapes thatbest encompass one or more identified objects in the image, where insome situations the regions or shapes can overlap. As describedelsewhere herein, the objects can be identified using object recognitionalgorithms (e.g., edge detection algorithm, Harris corner algorithm,etc.), and based on the size and/or shape of the object, the image canbe segment into portions that encompass those objects. In the situationwhere one or more objects overlap, the associated region and/or shapemay overlap. In various embodiments, the user may indicate an object ofinterest in the image such as by “tapping” the object displayed on thescreen, tagging the object, or otherwise indicating the object ofinterest. For example as illustrated in example 360 of FIG. 3( d), theuser has indicated the tree as the object of interest. In thissituation, a shape encompassing the tree can be used as the imageportion. The service can search the database for similar sized patchesto determine an unimproved patch matching the image portion.

In some embodiments, each image patch can be part of a respective imagepatch group. For example, each image patch group can include multipleversions of the image patch, where each version has undergone adifferent enhancement. Additionally or alternatively, the image patchgroup can include related or similar patches, where the patch can be ofthe same object captured from different viewpoints or of similar objectsthat share at least one or more common features. For example, FIG. 4illustrates an example group of image patches useful for imageenhancement. As shown in FIG. 4, an image 408 captured by a computingdevice has been segmented into one or more portions, and objects 420,422, 424, and 426 have been identified in the portions. As described,one or more of the image patches (or groups of patches encompassing anobject) can be part of an image patch group. For example, patchesmatching the image portions that make up object 420 can be part of animage patch group 430, where the image patch group can include one ormore versions (432, 434, 436) that make up the image patch group. Thenetwork service can perform image patch matching to determine whichversion is most similar to the image portions. As described, imagepatch/portion matching can include comparing one or more pixels of afirst image portion (e.g., source image portion) to one or more pixelsof a second image portion (e.g., library image portion) to, for example,determine a similarity in at least one image aspect (e.g., pixel color,pixel hue, pixel brightness, etc.). If one of the library image patchesmatches the image portion within an allowable threshold, the networkservice can modify the image portion using the matching library imagepatch such as applying image modifications made to the enhanced libraryimage patch to the image portion or by merging the enhanced image patchwith the image portion.

In some embodiments, the example image patch group can be an image patchpair comprising a lower resolution image patch and a higher resolutionimage patch (i.e., lo-res-hi-res pair). Accordingly, in order to enhancethe image, the network service can, at least in part, modify the imageportion using the image patch and/or the higher resolution version(s)(e.g., of the image patch). For example, the image portion can bereplaced by, merged with, or otherwise modified using the image patch ifthe image patch has a better image quality (e.g., resolution, sharpness,brightness, color, etc.) than image portion.

In various other embodiments, the matching of image patches to imageportions can be processed in an increasing order of image patchresolutions. For example, lower resolution image patches can be searchedfor first to identify patches that match a given image portion.Continuing with the example, subsequently, higher resolution imagepatches can be analyzed to determine whether or not they match the givenimage portion. Further, even higher resolution patches can be comparedwith the given image portion to check for matching. The network servicecan select the highest resolution image patch that still matches thegiven image portion within an allowable threshold. The selected highestresolution image patch that still matches the given image portion can beused to modify and enhance the given image portion.

FIG. 5 illustrates an example process for enhancing an image inaccordance with various embodiments. It should be understood that therecan be additional, fewer, or alternative steps performed in similar oralternative orders, or in parallel, within the scope of the variousembodiments unless otherwise stated. A user of a computing device (e.g.,a mobile phone, a tablet computer, etc.) can use the camera to capturean image of a scene or subject matter(s) of interest. The image can betransmitted over a network and received 502 at a network service (e.g.,networked image service) operating on a remote server for processing.The image can be transmitted along with metadata associated with theimage (e.g., data tagged with the image, information related to theimage, etc.). As described, the metadata can be obtained by sensor(s) ofthe device can be stored along with the image. For example, the metadatacan include (but is not limited to) geolocation information atsubstantially where the image was captured, a direction at which thecamera of the device was pointing while capturing the image, a time ofday/year when the image was captured, data inputted by the user toprovide one or more details about the image, and/or other informationrelated to the image.

The network service can communicate with a database that includes one ormore image and/or patches of images. As described, the images caninclude entire scenes and/or objects, or portions of those scenes orobjects. The images can include images that have been enhanced throughimage enhancement processes, which can include, for example, adjustingeach of the images (or portions of the images) by one or more humanexperts or other automated image editing processes. The adjusted patchescan be linked to the original unimproved sections of the same image.Additionally, in many embodiments, the procedure by which the sectionswere modified can be stored with the sections.

Upon receiving the image, the network service can compare 504 each of aplurality of image portions of the image against a library of imagepatches, each image patch of the library of image patches representingat least a portion of at least one object and having undergone at leastone image modification, to locate 506 a matching image patch of thelibrary of image patches that is determined to match a matching imageportion of the plurality of image portions of the image. For example,for the image portions for the image, the network service can search forunimproved patches that substantially match (i.e., matches within anallowable threshold) a respective image portion. If a matchingunimproved patch is found for a respective image portion, the respectiveimage portion can be modified using an enhanced patch that correspondsto the matching unimproved patch (i.e., the enhanced patch is in thesame patch group/pair as the unimproved patch). For example, the imagemodification, which was applied to the matching image patch, can beapplied to the matching image portion to generate 510 a second image(e.g., an enhanced image) including at least the modified matchingportion, the second image being an enhanced version of the first image.In some embodiments, the respective image portion can be modified using(e.g., replaced by, merged with, etc.) the enhanced patch correspondingto the unimproved patch that matches the respective image portion. Inother embodiments, rather than use the enhanced patch from the database,the network service can use stored procedure information from thedatabase for how the enhanced patch was modified and apply thatprocedure to the image portion. The procedure can be applied to theimage portion based at least in part on how well the image portionmatched the unimproved patch.

The enhanced image can be provided 512 to the computing device from theremote server, and the enhanced and/or the original image can bepresented to the user. The user can be presented a prompt on whether toaccept, save, or discard either image. The accepted (and in someinstances the discarded) images can be stored on the computing deviceand/or the remote server, such as in a database located on or separatefrom the remote server. In accordance with an embodiment, the imagesstored on the remote server can be made accessible to the computingdevice. For example, the user may have created an account with a servicethat provides a cloud-based photo album, and the user can decide tostore certain images in the cloud-based photo album.

What is claimed is:
 1. A computing device, comprising: at least oneprocessor; and memory including instructions that, when executed by theat least one processor, cause the computing device to: receive a firstimage captured by an image sensor of a client device; compare each of aplurality of image portions of the first image against a library ofimage patches, each image patch of the library of image patchesrepresenting at least a portion of at least one object and havingundergone at least one image modification process; locate a matchingimage patch of the library of image patches that is determined to matcha matching image portion of the plurality of image portions of the firstimage; locate the at least one image modification process associatedwith the matching image patch in a database; apply the at least oneimage modification process associated with the matching image patch tothe matching image portion to generate a modified matching imageportion; and generate a second image that includes at least the modifiedmatching image portion, the modified matching image portion havingundergone the at least one image modification process, wherein thesecond image is an enhanced version of the first image.
 2. The computingdevice of claim 1, wherein the instructions when executed further causethe computing device to: determine a second image modification processapplied to a second matching image patch determined to match a secondmatching image portion, wherein the second image modification process isdifferent than the at least one image modification process; and applythe second image modification process to the second matching imageportion.
 3. The computing device of claim 1, wherein applying the atleast one image modification process further includes performing atleast one of merging the matching image patch of the library of imagepatches with the matching image portion of the plurality of imageportions of the first image, adjusting a color of the matching imageportion to match the color of the matching image patch, adjusting a hueof the matching image portion to match the hue of the matching imagepatch, or adjusting a sharpness of the matching image portion to matchthe sharpness of the matching image patch.
 4. The computing device ofclaim 1, wherein the instructions when executed further cause thecomputing device to: determine a lens type and an image sensor type ofthe client device based at least in part on metadata associated with thefirst image; and apply one or more image processing algorithms to thefirst image to remove at least one image artifact caused by one of thelens type or the image sensor type.
 5. A computer implemented method,comprising: obtaining an image; comparing one or more portions of theimage against a library of image patches; determining a matching imagepatch of the library of image patches that substantially matches amatching image portion of the one or more portions of the image;identifying at least one modified image patch associated with thematching image patch; locating, in a database, image modificationinformation for the at least one modified image patch; and modifying thematching image portion, using the image modification information for theat least one modified image patch, to alter at least one aspect of thematching image portion.
 6. The computer-implemented method of claim 5,wherein the determining step further includes: comparing a set of pixelsof the matching image patch to a set of pixels of the matching imageportion, wherein the comparing utilizes at least one of line detection,edge detection, feature detection, image color comparison, image texturecomparison, image shape comparison, or image distance measurement. 7.The computer-implemented method of claim 5, further comprising:processing metadata associated with the image, wherein the metadataindicates at least one of a geolocation of a computing device at a timewhen the image was acquired using a camera of the computing device, adirection at which the camera was facing at the time when the image wasacquired, a temporal aspect of when the image was acquired, a subjectmatter associated with the image, or a description associated with theimage; and selecting a subset of a plurality of library image patchesbased at least in part on the metadata.
 8. The computer-implementedmethod of claim 5, wherein the modifying step further includes: mergingthe matching image portion with the at least one modified image patch toalter the matching image portion, or determining at least one imageenhancement process applied to the at least one modified image patch;and applying the determined at least one image enhancement process tothe matching image portion to alter the matching image portion.
 9. Thecomputer-implemented method of claim 8, wherein applying the determinedat least one image enhancement process further includes: determining oneor more neighboring image portions to the matching image portion; andapplying the determined at least one image enhancement process to theone or more neighboring image portions.
 10. The computer-implementedmethod of claim 8, wherein merging the matching image portion with theat least one modified image patch further comprises: performing at leastone of averaging pixel values for corresponding pixel locations,weighted averaging pixel values for corresponding pixel locations, orinterpolating pixel values for corresponding pixel locations in thematching image portion and in the at least one modified image patch. 11.The computer-implemented method of claim 5, wherein the determining stepfurther includes: determining a subset of the library of image patchesstored in a database based at least in part on metadata included in theimage, wherein each library image patch of the subset of the libraryimage patches is associated with one or more modified image patches; andsearching the subset of the library of image patches to determine the atleast one modified image patch that substantially matches the matchingimage portion.
 12. The computer-implemented method of claim 11, whereinat least a subset of the library of image patches has been modified by aperson.
 13. The computer-implemented method of claim 5, wherein the oneor more portions includes at least one of one or more square portions,rectangle portions, triangle portions, regions, or region shapes. 14.The computer-implemented method of claim 5, further comprising:providing a processed image and an original image to a computing devicefor display on a display element of the computing device, the processedimage at least including the modified matching image portion.
 15. Thecomputer-implemented method of claim 5, wherein the steps of obtaining,comparing, determining, identifying, locating, and modifying areperformed by a network service located on one or more networked serversin communication with a computing device.
 16. A non-transitory computerreadable storage medium storing one or more sequences of instructionsexecutable by one or more processors to perform a set of operationscomprising: obtaining an image; comparing one or more portions of theimage against a library of image patches; determining a matching imagepatch of the library of image patches that substantially matches amatching image portion of the one or more portions of the image;identifying at least one modified image patch associated with thematching image patch; locating, in a database, image modificationinformation for the at least one modified image patch; and modifying thematching image portion, using the image modification information for theat least one modified image patch, to alter at least one aspect of thematching image portion.
 17. The non-transitory computer-readable storagemedium of claim 16, wherein the modifying step further includes: mergingthe matching image portion with the at least one modified image patch toalter the matching image portion; or replacing the matching imageportion with the at least one modified image patch to alter the matchingimage portion, or determining at least one image enhancement processapplied to the at least one modified image patch; and applying thedetermined at least one image enhancement process to the matching imageportion to alter the matching image portion.
 18. The non-transitorycomputer-readable storage medium of claim 16, wherein at least some ofthe one or more portions are included in a library that includes thelibrary of image patches and are utilized as library image patches, andwherein the library is hosted on one or more networked serversconfigured to communicate with at least one network service.
 19. Thenon-transitory computer-readable storage medium of claim 16, wherein thelibrary of image patches are obtained from at least one of a socialnetworking service, a networked image storage service, or a computingdevice used to acquire the image.
 20. The non-transitorycomputer-readable storage medium of claim 16, wherein the modifying stepfurther includes: determining a resolution of the one or more portionsof the image; and adjusting the resolution of the one or more portionsof the image to match a resolution of at least one patch of the libraryof image patches.